Skip to content

docs(research): Measure which Harness tools survive an ACP attach - #788

Merged
omesser merged 4 commits into
mainfrom
spike/668-harness-tools-under-acp
Sep 17, 2026
Merged

omesser merged 4 commits into
mainfrom
spike/668-harness-tools-under-acp

Conversation

@omesser

@omesser omesser commented Sep 16, 2026 •

Copy link
Copy Markdown
Owner

Why

#668 asks whether a Harness attached over ACP keeps its tools or is reduced to chat. docs/research/harness-native-tools-under-acp.md (#669) answered from adapter source and vendor docs and left five items open because it ran nothing against a live Harness. Two of those claims matter most: which tools the Agent SDK session lists, and whether cwd decides which of the user's MCP servers load. Both are cheap to measure, and #669 measured neither.

What changed

Two files.

docs/research/harness-tools-under-acp-probe.md is new, anchored at 39fd012c. It records live probes against claude-agent-acp 0.78.0 (Agent SDK 0.3.270), grok agent stdio, hermes acp, opencode acp and pi-acp, with every claim labelled Fact, Inference, or Assumption. It carries the eight answers, the per-Harness matrix, the ranked options with their ADR-0003, ADR-0018 and ADR-0023 consequences, and the smallest probe left to run.

docs/research/harness-native-tools-under-acp.md gains one "since shipped" preamble paragraph and nothing else. Its claim 14 cites a 20-second turn budget, which was model::TIMEOUT at its anchor e070216. #695 gave a Harness turn harness::TURN_TIMEOUT of 120 seconds (src-tauri/src/harness.rs:71 at 39fd012c, for #690). Per docs/agents/docs.md an anchored note keeps its record and gets a preamble line when a finding has since shipped, so the 20s prose below it stays as written.

Measured, off the Agent SDK init message through the adapter's emitRawSDKMessages extension:

  • A claude attach keeps 33 built-in tools, WebSearch and WebFetch included. AskUserQuestion and computer-use are the only absences.
  • User-scope MCP servers and claude.ai connectors load regardless of cwd. A local-scope server loaded only when cwd was the directory it was added in. That is the cwd claim docs(harness): Research which Harness tools survive an ACP attach #669 inferred, now observed.
  • No computer-use server appears, with or without the undocumented ALLOW_ANT_COMPUTER_USE_MCP=1. Anthropic's page names four gates, and this machine's Team plan trips one before the interactive gate can be observed, so that cell stays an Assumption with a probe filed for it.
  • Grok lists 33 tools by self-report, web search and shell included.

Round two (2026-09-17) closed the rows round one could not reach, all on this Mac.

  • hermes: 27 tools. Measured through its own /tools slash command over ACP, which needs no model call, so the dead local provider stopped mattering.

  • opencode: 10 tools (bash, edit, glob, grep, read, skill, task, todowrite, webfetch, write). Reached by overriding the configured model to xai/grok-4.6 in a copy of the config; this machine has XAI_API_KEY. webfetch yes, no web-search tool, which matches the vendor's "conditional".

  • cursor-agent: 19 tools and codex: 22 tools. Both are named rows in harness.rs that round one's matrix did not cover at all. codex is the only Harness besides grok listing an ask-the-user tool.

  • pi: 4 tools (read, bash, edit, write). Its local model server was the thing that was down; once it was up, pi answered. The pi v0.85.1 the earlier note recorded is a session banner printed before the turn, not pi's answer, so the tool list is on the last line of AGENT_TEXT rather than the first. Measured twice with different prompts, identical both times, and pi --help opens "pi - AI coding assistant with read, bash, edit, write tools". Claim 17 goes Assumption to Fact: pi keeps its whole documented set under attach. Separately, pi-acp advertises mcpCapabilities: {http: false, sse: false}.

    That also gives a rule worth keeping: a probe that returns only a banner is evidence about the provider, not about the Harness.

  • Claim 7 is now a Fact. A fixture directory holding only a .mcp.json put probe-project-scope into init.mcp_servers. Project scope keys off cwd, measured rather than inferred.

One correction, and it re-ranks an option. Round one inferred that advertising elicitation.form switches on exactly one tool. Measured with clientCapabilities: {"elicitation":{"form":true}}, init.tools is the same 33 names as with {} and AskUserQuestion is in neither. The adapter's own comment says why: AskUserQuestion arrives as a normal permission check, and the capability re-renders it as a form. It recovers no tool. Ranked option 2 said "it is the one client capability that switches a tool on" and now says what it actually buys, which leaves option 1 as the only tool-recovery option in the list.

Also refined: hermes lists all ten browser_* tools while the start-up CDP check marks them unavailable for the turn. A listed tool is not a callable tool on that Harness.

Every measurement agrees with the merged note's Facts. The note also records two corrections to what the repository says elsewhere, neither edited here. ADR-0003's Consequences name one of four computer-use gates, and README.md:159's "Chat-only (no MCP)" reads as a claim about pi's own tools.

By design this PR has no product code, no README or ADR edit, and no _meta.claudeCode.options hatch in the product. The probe client is embedded in the note so a reviewer can rerun every measurement without ai-buddy code.

Follow-up issues filed on V1 for the options that survived: #782 (attach cwd), #783 (elicitation.form), #784 (README tool classes), #785 (ADR-0003 consequences), #786 (computer-use probe on a Pro or Max Mac).

How to verify

Rerun the two Claude probes from the note's last section with two cwd values and diff INIT mcp_servers. On this machine the second run added caveman-shrink and nothing else changed.

Round two's three techniques are written up in the note's new "Round two" section with runnable commands: /tools for hermes, an OPENCODE_CONFIG model override for opencode, and a .mcp.json fixture directory for claim 7. The elicitation.form probe is a one-line change to the client, also shown.

pre-commit run --files docs/research/*.md passes. node --test tests/*.test.js 238 pass, 0 fail.

Notes

Does not close #668. The owner accepts the spike.

— Claude Code agent, on @omesser's behalf.

🤖 Generated with Claude Code

@omesser
omesser force-pushed the spike/668-harness-tools-under-acp branch 2 times, most recently from 7c46acd to e93e5d9 Compare September 16, 2026 21:06
omesser and others added 2 commits September 17, 2026 02:19
Live probes against claude-agent-acp 0.78.0, grok, hermes, opencode and
pi-acp, recorded with Fact / Inference / Assumption labels. Companion to
the desk research in #669.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Prose only. Every label, citation, table row and verdict is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@omesser
omesser force-pushed the spike/668-harness-tools-under-acp branch from e93e5d9 to ad055e2 Compare September 16, 2026 23:23
omesser and others added 2 commits September 17, 2026 02:44
Provider-free /tools for hermes, an xAI model override for opencode, and
the two named rows the matrix never had. Corrects the elicitation.form
inference: it re-renders a question, it does not switch a tool on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The `pi v0.85.1` two earlier probes recorded is a session banner, not an
answer. With the local model server running pi lists read, bash, edit,
write, which is its whole documented set.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@omesser omesser left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CLEARED @ 6325d56 — docs-only measured companion to #669 / #668.

Spec (#668 Done-when)

  • Research note with Fact/Inference/Assumption table ✓
  • Per-Harness matrix (7 rows, measured where possible) ✓
  • Ranked options with ADR consequences ✓
  • Smallest falsifying probe for top option (cwd + Pro/Max computer-use) ✓
  • Follow-ups already filed for surviving options (#782–#786); listing them on #668 next

Standards

  • Docs only (docs/research/harness-tools-under-acp-probe.md + “since shipped” preamble on the #669 note). No product code.
  • Honest corrections (elicitation.form does not add a tool; turn budget 120s; self-report vs init caveats).
  • CI green; MERGEABLE/CLEAN.

Soft: tip may need rebase before land if main moved. Ready to merge when you say yes.

— Cursor agent (Architect), on @omesser's behalf.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

spike(harness): Keep the Harness's own tools under ACP (computer use, web, user MCP)

1 participant